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^ABSTRACT . . 

A useful, values^oriented human resource development 
(HRD)' program* evaluation* approach "is onje that is empirical with its 
orientation derived* from a s^ocial/teqhnic^l perspective. Ihe optimum 
mix bet^€fen the technical skill"S of evaluation' and the required 
social skills of ^ejtaluation delivery is crucial for the guaiity 
ass^tance procfs^. T^e core to this approach is groundied in the 
int6rrelat|.onship of thr^ee cbnti^ua: (t| th^g^^^neral context of ^ 
training, education, and development; ^ (2) time dimensions^ and^ (3) 
actiba companents suclii as souxc^e, typ^s, and collection methods for 
data. The most critical dimension , of thfe socio-technical continuum 
approach is time. |i geneial f rameirork- can be provided to indicate the 
initial considerations and the immediate.i" intermediate, 'and long-term- 
positions. ' The direction • for ^tevaluation design and understanding of 
evaluation results should be considered under the initial 

ideratibns ar^a tyf means o£ jevaluation iss-ues, participant and ^ 
management exp'ectations, and the prelected results on tuture 
decision-making- Th6 day-to-day developiaent .and mana^gement of quality 
' assunance tasks focus 'on the levels of information, sources of data, 
i^Tlnd methods *o£ collecting data. Evaluating the evaluation can be , 
achieved through* the following \technical and social components: 
reliability", validity, utility, credibility, and feasibility^ This 
type of analysis T.inks evaluation to planning and thus to recycling 
and feedback vhile also b'lending judgment irith objective evidence, (k 
16-item re^ferenc^e section is included.) (BPS) 
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^ . , A Socio-Technical Continuum Approach 
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/ ABSTRACT ^ • 

^The intent of this paper is Co conceptualize an approach to training * 
program evaluation (specifically, HRD quality assurance) that would be useful 
in business and industry settings. This approach is- a blend of evaluation ^ 



theory, experience gained from evaluation practice and e^irically-derived 

\ 

concepts. Significant stress is placed on t-he relationship between the 
technicai skills of evaluation research and the required social skills of 
evaluation delivery. An optimum mix of thesfe skills is viewed as crucial 
for the quality assurance process, to^gain' credibility with decisiort makers, 
jpspecially management, as well as to achieve greater utility for the entire 
evaluation effort. 

The key topics discussed include time as a controlling dimension, the 
role of issue vs. purpose in the quality assurance mission, the use of level 
of information and utility of evaluation. results. To show the interrelation 
ships among the variables of time, type of program (training, education or . 
development), issues, 'sources,* methods and lev,els of dnf orraatlon,^a matrix*, 
sterna is outlined to illustrate that the nature of thinking involved in 
designing quality assurance processes is configural and not linear. 
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Quality Assurance of Human Resources Development Programs: 
A SoCio-Techoical Continyum Approach 



Introduction • . . 

Human .Resources Development (HED) program evaluation, wherever it is 
seriously employed, is a difficult and time-consuming process. ' Co^npared to 
an experimental design investigation, a typical HRD program evaluation ' project 
generally appears *to contain more variables, %ave less control /md offer fewer 
definitive results and implications. The inherent complexity and political 
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interactions that are an integral part of t;he program evaluatiofi process, 
deserve a great deal of credit for this 'disparity . • - , ^ 

• ' \ • • : ' s • ' ^ * ' . 

One of the* mofft serious o^these interactions can-be illustrated when 

/ ■ • ' " 

the- evalilator is encountered with the question, "Why evaluate*?" ^Thompson-, 

• , . . • - ' •• ■ * 

1978). The asking of this question alone indicates that the credibility 
of the evaluator apd the evaluation process is virtually nonexistent^ If • 
this question' is considered a legitimate inquiry, the pro'pos'ed evaluation 
would have to be outside the mainstream of planning, developing, delivering, 
and'managing HRD activities; this^ is a situation which should not be accepted 
nor tolerated. In order for evaluation actiyities to achieve and maintain 
credibility, evaluation must, be viewed as an integral segment of all HRD 
developmental and implementation efforts — its role being supportive as well 
asj judgmental . Thus, progr^ ^valuatioi^^ should never be viewed as a solution 
in^ search of^a problem. . ^ 

, , ; ■ ~ ■ . ' 

To- assist evaluation practitioners through the coranjonly eticountered 
maze'^f f)roblems, evaluation theorists have supplied numerous models (see a 
review 'by. S'feufflebeam and'webster, 1980) for consideration. Although, the 
model *p resented tin, this paper is*"admlt'tedly not unique, it does, offer a ^ 
different view. .*lt 'is also probably labeled bettefc as an approach than 



^Paper presented at AERA. Annual Meetlngv SI6; * Training In Business and Industry, 
Los Anseles, CA^>/^Aprll,.lj, 1981. ^ \ O ^ ' 
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a model, and its application is directed specifically ati quality assurance < 
of HRD programs (genericaily , program ^valua'tion) . The basis fpr the 
approach is empirical, and its orientation is derived from a socio-teohnical 
systems perspective. For' its core, this' approach posits three interlocking , 
and interactive matrices to demonstrate the interrela.tioriship of^three 
continua: 1) the general context of training, education, and development; 
2) time dimensions; and 3) action components: rypes of data, solirces of 
data^ and co-llection methods for data. Technical, political and social ♦ 
components are Interwoven' throughout tK^design. ^ 

General Context ^ ^ . * • . 

The general context of HRD programs in business and industry has been 

,^ 

subdivided by Nadler (1971) into training, education^ and development. These * ^ 

subdivisions^can be analyzed for the purpose of evaluation application as 

shown in Figure 1. If one views training, education, and development as a 

• ' ** - 

continuum of a people development /learning process, the corresponding types 

of objectiv'es go from concrete to abstract arid the length of the. HRD programs 

%o from definable to undefinable. Desired: outcomes move from direct performance 



Insert Figure 1 About .Here 
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related results under training to' career advancement to organizational growth 
for Development.' Relationships between the individual and the orRaniz^tlon ^ 
change f^om those primarily concerned with the individual to thos6 concerned 
primarily with' the organization. * Points- for ^decisionTmaking *are clearly . 
defined* for Training, projected ^within *a certain* time period for Education, 
but can only ie classified as future-oriented^ for Develppment . 

The nature of these three sets of activities (If a ining, . Education, and^ • 
Development) therefore, has some direct rel^v^oe to the design of qaality . 
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assurance programs. The continuous nature also implies a close, wdtking 
relationship between the planning of HRD progr^inis and their Subsequent 
evaluations. A further implication is that particular evaluation .techniques 
used for Tra'ining programs would likely have minimal impact br credibility 
when used for Development prograifts. For example, a reaction questionnaire 
may have direct and important impact in the evaluation ^o^f Training, but 
substantially more depth-oriented techniques, such as interviewing, are 



likely to yield more impact for evaluation of Development activities. Time 

and amount of progr^ specificity, therefore, have. a relevant bearing on the 

\ ' ' 

levels of decision-making that will occur and the 'complexity of evaluation 
issues Ohat must be taken "into Account. ' - . 

T±me as a 'Controlling. Dimension . , .. ' . ' 

^ Probably the most critica!^ d'imensioh of this^ socio-technical continuum 
' \ ' * * • 

•approach to designii^rg quality assurance programs is time. While evaluation 
a*s an activity looks back, i.e., records what happened, its Unction is 
really, forward .looking. Evaluation has a requirement to reach beyond 
^ the present data and present choices among alternatives not directj-y studied. 
Purely objective-based evaluation tends not to give full consideration to ' 
unintended outcomes that may have important positive or negative effects on * 
organizational growth and vitality! When viewed longitudinally, all 
evaluation appears ^formative (CrxDnba^h, 1978); very few decisions as a result 

, * ' 6 

\ , 4 . 

of evaluation kre truly ultimate ones. * ' . • 

Time is also important in determining 'when to begin and end^ the evaluation 

^Efort; oftentimes evaluatipn seems not to start soon enough nor finish long 

enoagh after* tjie^ actual training expei^ience. Figur'e 2 provides a general ^ 

framework from which emanates a general context for evaluation design. • Thi^ 

framework was intended to be jgeneric, thus it maybe imposed upon any HRD 
• • * 

c .progi^am encountered. 



t Inserf Figure 2 About Here 

time dimension continuum 
are ^probably least^ritten about> but for different reasons. « Immediate and 
Intermediate components are most often described so further elaboration is 
not 'presented here. ^*L<!mg~term" objectives arc generally seen as'dif&icult 
to define and ^easure. For example^ improvement in organizational productivity 
is probably tog abstract of an objective to measur^ reliably especially if 
,we want to attribute a certain proportion of productivity gains directly to , 
training activities within anJiRD program. The' **Initial Considerations" area, 
on the other hand, can be easily utilized tu guide' the direction for evaluation 
design and tiie understanding of evaluation results. 

j ' " . . • ■ , ■ ■ ' 

^ VThree primary areas of concern need to be considered under the- "Initial 
Consideration" 'area: 1) evaluation issues, 2) pdrticipant -and management 

expectations, and 3) the projected results on' future decision-making . Of 

. ' .r 

prin^ary importance among these concerns are the evaluation i;5sues surrounding 
the quality assurance process. Evaluation issues are^ meant as a replacement 
for Tylerian evaluation purposes 'pr objectives* An issue is a central concern 
OE question such as: Were prpgram participan^:'s readily able to 4:oraprehend 
materials? or Did training instructor^ use effective teaching' techniques? 
important problem., with a purely'' obj^ectives-^based evaluation is the harrow 
preoccupatloQ with>specif ic objectives without regard to uhintend^ or, 

' • ; * . ' * / ' 

unforeseen behaviors and resultsJ * An issue can ii;iore easily guide ef fcirts and 

• _ , ^' ' •. • ''b ^ .•*■ '.: • 

may* be-brokeh dow);i into specif ic ^ components', 

"^It would be ^\ldf f icult.,' If not impossible, to conduct' a-use^ evkluatW 
if the designer was^ not knowledgeable about the particular organizational ^ 
Structure ajad have some genefral background in the pol;|.tics of intierpersonal 
behavior.* Knowledge of antecedent qonditions fpr perspective is a -mandatory 



/ ^5- 



requirement ta identifying the central ^issues in an evaluationJ By utilizing 
the "issue approach" ^to design, ..through contributions of polit:|cal informants, 
program observ6rs, relevant theory and personal experience, the designer is . 
better able to guide the effort around the common pitfalls and; tradeoffs^ 
usually encountered, Spme of these pitfalls include not considering all 
relevant decision-making audiences, focusing' on only minor aspects uf *the ^ 
program, or mor^ importantly^ liMted consideration of the^ut?.lity and useful- 

V ' ' ' '■ 

ness of evaluation results r rbs^does not matter whether URD program objectives 

r (• ; } ' ■ 

are tightly or loosely defined, the evaluation issues, as opposed 'to c^pjectives 
should form the basis 'of the quality .assurance mission. ' ^ 

r ' 1 O • 

' ^ ; - 

The seconH area of concern revolves%around /^audience" expectations prior 

* , - r ' ' 

to prograi^ activities ► "Audiences'* are usually those who, havje sonie stake 
the evaluation and prqgram such as the participants in. t]he program, the 
teaching ^taff, and the drganizational management! From the participants we 
need to know what they hope to learn, gain'or achieve 'as a result of the ^ 

. ^ ., , #- 

program. For. the teaching staff, historica-l bacTcground information on the 

participants -is helpful in establishing vhat the partitlpanta bring to^the' 

program (abilities, atditudes^ past performance). Asking the participants 

what they expect to learn or gain can be compared with opinions or^ actual . 

results after the program. Similar methods may^^b^^sed with instructional 

'staff and management. Obviously, these groups are looking at t'he HKD activity 

from different .viewpoints. ' For example, the straff may be interested in ^ , 

* * " * « 

acceptance -and usability of developed materials; management^ may be looking^ for 
"star*' performers or changes in .identification with the organization. 

Tl\e third area of. concern under "Initial Considerations", is with the 
•impact /of projected results oh f u'tiire activities and de9ision~making . Once, 
evaluation issues have been identified and i^xpec tat ions made explicit, it i-s ^ 
* time »to foraulate the evaluation design and do some hypoth^jMzi-ng wi,tl| respect 



■1 



• to potential' courses of action. This hypothetical **what-if** stage is crucial 
for evaluation management. After all, an evaluation ^design is meant to control 

-J . • ' . • . - ■ . . . 

potentdal and actual tradeoffs whiah minimize post-evaluation uncertainty witjiin 



a fixed coft. Concurrently, the ev^luator must be aware J? ftie community of 
.decision makers who/ await the results of the eva,luation. Taking these points 
into account will assist in giving direction to^ the effort and minimize the 
number of alternative explanations required to interpret the results. 
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Action Components 

Thus far, this socio-;>technical continuum approach to evaluation has dealt 

with the establishment of- the general context within which quality assurance 

takes place (Figure 1) 'and with the variable of time for past arid future 

functional directions (Figure 2). It is now appropriate to illuminate on 

those aspects of evaluation that have direct relevance to phe day-to-day 

development and management of .quality assti^ance- tasks . ^ The followin>^, 

description focuses on three rjajor interacting action components of 

ev^ilua'tlon design; levels* of infcJ^mation, sources of data, and met}iods 

for ^collecting data (See Figure 3)* . / 

. The first of these ;actior^ components, Levels of -information, may be 

> ■ 

unfamiliar to njost readers, s6 its definition and utility is given Initial . 

consideration. Levels of information are meant .to provide guidelines for 

» 

evaluation design and management. There exist a* number of definitions for 
tWese "levels," all of them implying similar interpretations, 'information 
ma^ be VieWed in two cor^tinua — 1) in terms of focus: from a general judgment 
to descriptive behavioral information, of . 2) in t^rms of range: from global 
information tcr very specific inf ormatip.n. In the first continuum, a general 



opinion impdiesa judgment of overall worth — summative evaluation if you 
Willi, description Implies observatlcTis, reporting of facts- or specific 
behavioral reaction — thus., basic formative evaluation^ Consider for example,. ,^ 



Jfhe evaluation of' teaching staff performance .in a ^giv^n HRD program^ If the 
evaluation "issue is how can* we improve delivery skills of instructors, one 
would not be very interested^ in^ the participants' answer to i:he .question: 
How would you rate this instructor ^s* overall teaching ability? This question 
is reaily-a global question,^ o\te requiring a statement' of worth (a general 
judgment) • It may be an important question, but it is not directly relevant 
to the issue of delivery skills ♦ Instead, 'questions such as:" Was the pace 
of presentation, too fast or too slow?. Did the instructor explain new ideas 
by relating them' to familiar concepts?, or Oid the instructor have araoying 
mannerisms which detracted from delivery?, would yield information mure 
directly relevant to *the issue ;of delivery ^skills. » * 

• For the sake-,of simplicity , and utility, we may arbitrarily divide the 
continue (t/hlch are essen;:ially different terms with the same meaning) of ^ 
glob^^ to 'specific (Smock and Cro6ks, 1973) or* jitdgment ttj description 
(Feldman,, 1977), into three categories — global (Levels), general concept 
(Level II), and sjieclf ic, (Leyel III)* Global judgmental information /Level_ I) 
is primarily directed at ^forming a generad opinion o£ HRD acfivi.ty success 
or failure, i.e.; was the workshop successful, did the participants learn 
anything, did the teaching staff perform well, or how did it go. Global 
information has primary utility in making comparisons across individuals, 
groups, or settings • These comparisons may be absolute > relative or both, 
but experience suggests that relative comparisons would probably ^ield the 
most fruitful results. If global questions are* appropriately applied, they 
should have some significant relationship with the general concept and specifc 
information. . • ' ' * 

General concept information (Level 11) by nature of its in-^be.tween status^ 
has some characteristics in ^lommon with bqth global and specific information. 
The primary purpose, however, of general concept information is to identify 
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general strengths and weaknesses within the evaluation issue^being investigated. 

Returning to the example of the teaching' staff member, questions sueh as — 

Was the instructor a good s-peaker?, Pid the instructor appear prepared?, ^ 

Were the discussions well led?. Did the instructor stimulate participant , 

interest in the subject? — attempt to diagnose^ lightly the potential . 

concerns with speaking ability, preparation, leadership, and stimulation, j 
>> ' ' * , ' 

respectively. If one closely examines information collected in most 
evaluation report^, a majority oT it can be Classified as Level II. An 
advantage of Level II information over Level J, is that both absolute and 
relative comparisons may be made with equal effectiveness tyider most conditions, 
Comparative vises of such data should be performed c-aut iousljj^jaying particular 
attention to the relevance and appropriateness^ across the groups of interest. 
Level- II inf oration lends itseJf well to making some absc/lute interpretation 
of results (better in this respect thaa Levei I usually), because it has a 
definable base of g^eneralization; however, it still lacks, direct knowledge 
of behavioral manifestations necesgary ?or specific diagnosis. The latter 
role is fulfilled by Specific or'*' Level III inf orjnation. * 

The hierarchical 'steps of, identifying a general issue (Global or Level I 
'information), followed by identification of components of the issue (General 

4 

Concept tr Level IIjLnformation) is furthei: .illuminated and •pin-pointed by 
»th^ most diagnostic information^ Specific or Level IIJ. In the example of * 
the teaching staff member,'* quest ions about presentation pace, clarity of 
new ideas and annoying mannerisms (given above) were illustrations of Level III * 
information., ^Oftentimes the nature of Level III informatton does not lend . . 
itsfelf well to standard or structured questions; answers may be better collected 

with "sof ter" techniqjues of evaluation such as specific ^pen-ended questions, 

- r 

interviewing or videotape analysis. This nature also 'does not permit useful ' 

O - - • 

gj^Q) comparative, inforraa^tion. Since the questions are specific (here for a given 



fstaff person in a give« teaching <^etting) , they are hot meant for generalization 
and ^ny cc^p^msons may ap^ally J>e misleading as we,ll as inappropriate. A 
potential disadvan^^a^- of specific Lfevel III information is that it mar bear 



little ot noeause and effect relationship with Level I informa'tiiTtt^ For 
exampl^,^ there is no guarantee that once ^ instructor eULniinates^-^nnoyijig 
luarrfeerism^l, overall teaching ability'would show marlce^d .ImprovemeiSl^'. * 

It is worthwhile' to note that this hierarchical system, .>rhile conceptual^ 
hfis been applied in rhe development of evaluative designs CSmoMc and cWoks, 
1973; Brandenburg, '1977) . Recently J oiSr inVestlg^Jtion of cpllege student 
ratings items (Brand en bur g^.Derry & Heng^^tlef, 1978)'jia^ yielded an^ empici^cal 
demonstration of the existence of this cuntinua of information .(aithoug-h w5 
Showed some justification for four levefs, this shtnJs the ar-bitrary division 

. ^ ' S ■ ^ .^i' . . • ^ 

i?nto three JLevels)-. Id summary, t^e concept of,;levels 'of information tjhoviTd 

' . ' ^ - ^ . :^ , • ■ 

be viewed as a heurisXJ^-.tool for guiding total ev^Iuatid'jt procfelss. * This 



toncept has dxrect .implications for> choic'e of . evalu^t i'^e thodcJlogy , * f or • 
deciding to wh^m to report infoi?matibrf; for interpreting results, Vartd, for , 



generally organizing thoughts f fom *£i pr^gticai perspective. - . # . 

The other two action components of cvaluati-on. design are the sources ' .r* 
o£i information 4nd l^he methods for cc^llectingT'information.- \^lle the two are 
separate, th^y are often confjised in the tr^i'de literature (for example, Alden, 
p. A9,, 1978rKirkpatrick,' M78) . Specifically, Kirkpatrick's popular fouf- 
component mO^el of Reaction, Learning, Behavior , and Results rends tcj over** 
simplify component interactions. In an^yzing Kirkpatrick's'^(19f8) explanation.^ 
Reaction implies one source arid maybe 6ne or two methods; Lciarning impliea- 
one source and on^ or two itfethods; Behavior Measures in^Qlude potentially four 

sources but undefined methods; R'esults me^ures appear to imply numerous 

• * ^ ' . . ♦ 

unidentified sources, specif ies -one method and inaugurates a hunt for criteria, 

Whrat Kllrkpatrlck advocates "in his model desrerves important ^consideration, ^ 

■ ■ ■ '■ ,11 
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however, by mfxing..soucces and metlicAis under one label 'along a single 
contihuup, a-user is likely to b© daught in ^ome^ of tTid pitfalls discussed 
earliet, ^ One* also gets the impression that an evaluatpr carries ciut* 
Reaction, Learning, .Behavior, and Results regardless 6i the HRi:(, program 
situatibn or ^activities* The life of an evaluator is .not that simple. 
Unless the methods for collecting information and the levels^of iaife/maixon ^ 
rre^ed are. defined and integrated i^ith sources of "suclf data, evaluation ^ 

e-redibility ^'ttd utility are -very likely to suffer.. Ench method and* source . 

"•^ . i 

. ' ' . * * • ' c 

should be justified in a congruent^ sense against -tho evaluation issue -i^ / 

eyaldators have always been knoyn to^ have -the capabilitry to. prcxduce an 
'o ♦ ' ' < * ^ ' . ' 

•abundance of useless information. Ef^ciency, cost-ef f ectiveriess' and' - ' 

mission sl^ould be emphasised; Kirkpdtrick*"s model .is not ^ys^em^^.ed . 



enough nor comprehensdyer enough to Satisfy thesf crit>r-tiK 
^ . ^, • . * ^ - ^--^ 

' " ^ ^ , ^-^"^ . . ^ * 

* Figure ,3 contains an an^ilvtical'frame^cOrlc- Ca ^et of interlocking* 

• ~ . ^- ' ' ■ ) ■'. ' • ' * 

• ^ -ma-trices)^ fQt the^cLif f^crentiatron betw^n.^soVirces and methods and <heir 
' *. ijoint interaction with eValuatioin, issues aifd levels gf in formation, tlach 
pf tfie twelve boxes in f^lgura 2 pbtent:^ly couid be filled with the 
/ * subsysteni of Figure 3. ^ the, listing* q£ sources in figure 3 is not jn^ant \ 



have^to be iifclu^ed.in investigatipg 



to be,,*4ll Inclusive, nor woii Id .111 «Wurces 

/ • I • . ' ' ^ ' 

every evaluation issue* The number of data collect^ion methods shouTd be 



/ 
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v>^ Ihseijt Figure 3 About, Here 



ydllwed to vary .depending^ on numerous conditions, notably the/issue 'involved 



and the naturet^f the information source ,.The data collection metl^od Column' 
may include s6ffie of the following': ^ ' * « ' / ^ ^ 



• * elxpectation statements. 

• objective questionnaire or survpy *, 

• "^open-^na^^questlcpnis ^or reactions. , :^ 

• ^ cognitive p^ey and pencil mea^ytes ,\ 
.•^ p&f ormande meas^ifes -,^ry / * 



4f > 
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observation and anecdotal records ^ * 
videotape of instruction^, 

structured interviews • . ^ * 

-previous evaluat:ion results ' , . 

demographic and Jiistorical dat^ collection " 

• unobtrusive, follpw-up techhiqjueff ^ • • 

* cost-benefit , analysis 

cost-eff ectiveneS^ analysis ^ *• 

longitudinal personnel data file examination , 
*" • ' , ' , 

Due to the nature of behavioral sclenc^, data. It is seldom wise tp depend 

I ^ . . > ^. • \ ; 

/ on one source and one method to yield valid cjohclusions. It is important 

therefore, td use triangulation (multiple method's-, multiple soui:pes of 

crass-checking and verification]^ among methods and* sources^ o .achieve validity 

and credibility.* Thus, one ^.s better off to use thr^e methods on one source 

or three sources with one method. Wit^(Ajt this type' of validity cross-checkin&, . 

the evaluation report is more likely to. suffer from a4.acrk of clear direction 

vhen it com'es ^to laying out alternative' 'courses of future action for decision , 

makers. ' " ' ♦ ' - ^ 



Evaluating the Evaluatign ^ . * 

Tliis apprcfach, which is meant to be dynamic and op^n-ended, * would not 
be complete if attention is not cjrawn ta the quality of Information collected/ 
The-five major categories of information quality and 'evarluation 3esign quality 
include the following ^Technical and social components; reliability, validity, 
.utility, credibility, and feasibility. -They are defined below. 

Reliability > Reliability is generally^defined' as the consistency 

of acquired inf ormation. Internal consistency measures (like i . c 

KR#20,or C9effic^ieTit Alpha) are appropria^te to use in. evaluating 
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the homogeneity of a group of questionnaire or test items* 
Discriminate reliability, like the Horst coefficient, is 

^ /'.important in evaluating Level I dafs^ for- making reliable 
-comparative distinctions. . Here stich coefficients must have 

S^kgh values. A third type of rellabirity, and. one, t}iat, should 
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be used more often, is that given by generalizability 
coefficients. Using general-Aahility theory (Cronbach, et al , , ' 
1972) one estimates certain variance .erbmponents for a ^ata 
collection scheme and can then generallze'over certain dimensions 
(or *^^cets) by modifying the sizes of particular design components.' 
A fourth type of reliability often associated with softer -evaluation' 
techniques 1^ replicability (Guba^ 1978). Cuba Suggests three i 
techniques for replicability: an audit '(independent judgments 
of- the same, data), overlapping methods (jested use of complementary ^ 
techniques to collect information. 'On ^the same^issue), and stepwise 
replication_(division of informati^on^A^'Ou^^.s^or data cuj^e|2j^n u^axas 
into random halves to undertake independent studies). 

^ Validity . Validity refers to a number of aspects of design 
and data quality exemplified by internal validity ^ external - 
validity, content validity, and , construct validity. Because 
these definitions are cpmmonly known "and applied, further 

i " • 

discussion of the potential biasing effects on the lack o^f valid 
information is not included here.' 'For softer evaluation techniques, 
Che, terms intrinsic adequacy^ replaces internal validity and ^extrinsic 
.adequacy replaces' external validity (Cuba, 1978). Intrinsic adequacy 
refers to such aspects as erecting safeguards against possible 
invalidating facts, establishing structural corroboration (use of 

. 4 

'triangulatlon and cross examination) and estaJblishing the credibility 
of findings, Extrins-ic adequacy is related to** generalizabriity 
through areas such as representative situations, ' timeliness, special 
^bject samples, and recurring encounters (see Cuba, 1978) • ^ 
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Utility . The results of the evaluation endeavor must be viewed 
as useful to the organization; the iufonnation leading to . the 
results tieeds to be accessible to decision makers, and .the 
information must be understandiiOTfe td them: Utility has both a 
, social and' technical s^ide which is explained in the next secti.on. 
Attention ;'to. data i*eporting formats should be given early on in ^ 
the evaluation pi^ocegs to influence readability and in&erpretability^ 
of the findings. 'If a deci3ioa* maker ^ has to ask — wheat's the point, 
xrnpw what — the utility of the ' information is^lost. 

Credibility . edibility of infomation and design Goncerns 
both the sources of d^ta and the -dec^'sion makers. To a certain 
„ extent it reflects the notion of fac^ validity ' — ■ it must be seen 
as important oo: relevant 'or it is not worthwhile.- Credibility 
for the-evaluation pfll'acti tioner must generally be? earned. It can 
be gained if those impleraeating a quality assurance program remain 
impartial, accurate and respect the perogatives ot the individuals 
involved as well as advocate and protect the confidentiality of tlie *\ 
data base- 

Feasibility , A comprehensive quality assurance program, while 
meet^^ most of the previous considerations, may not be possible 
due to lack of time, financial or pfiysical resources. Return on 
investment must be paramount when discussing alternative processes. 




^ Suimnary and Discussion ' ' '* 

. "\ * 

The purpose of this section is to.-sumraarize sorme of the key elements 

«* 

of theyframework presented and to integrate them with the social/ technical 
perspective advocated. , ' 
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A first general, concern Is how the approach presented here may be 
classified among those that have been oir are currently being used in the 
g-eneral area of evaluation. A recent article by Stuf f lebe^m ,and Webster ^ 
(1980) provides a good deal assistance on this point. They define an 
evalud:tion study as "one that is designed and conducted to' assist some 
audience to j^dge and improve the worth of some educational object"' 
(emphasis added, p- 6). They then ^cla«sif^'e'a'''srudTes as "^Mitidally-- 
Oriented," "Questions-Oriented," or "Values-Oriented." The present approach 
may be best classified as "Decision-Oriented," a subheading of the Yalues- 
Oriented stupes. Stuff leteam and Webst'er- charactetize decision-oriented 
^udies as those emphasizing ''that evaluation should be used proactively 
to help improve the program^s well as retroactively to . judge its worth" 
(p. I2).^ In comparing a number of evaluation strategies, these same authors 
characterize each strategy according to certain components. The components ' 
and dealriptions for decision-oriented 'studies are presented/ below (p. 16, 17) 



Description ' 

Decision situations ' - 

To provide a knowledge and ^ 

value base for making and 

defending decisions ' ' 

Decision makers (management^ 
clients, participants, staff) , 
their oonsituents and evaluators 

How' sh6uld a given enterprise be 
planned, executed, and reqjrcled 
Dn ordfer to foster human growth 
and- development at a reasonable - 
cosj:? 

SuirO^eys, needs assessment, case 
studies, advocate teams, ' 
observa tions , and . quasi- 
experimental and experimental design 

Cro|ibach, Stufflebeam , 




Component 
Advance Organizers 
Purpose 

Source of Questions 
Main Questions 



Typical Methods' 



Pioneers 
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A couple -of key potnt§ deserve ^phasis. First, the' evaluator is seen as' 
taking an active role in the total, evaluation process Xpr ^proactive** after 
Patton, 1978). Thus, the evaluator Soes not function as a disinterested 
observer. Second, in order to maximize utility of the evaluation, the ^ 
evaluator must specify alternative* decisions and must haye gathe^€^3" sufficient 



information to defend these alternatives. Thii:d, the.v<3eplsion-oriented .approach, 
like the approach described in this paper, provides .information on the continuum 
from judgtfient of overall wor'th to program improvemehj: generally useful for 
^"foster Cing) human growth and development at* a reasonable cost" (a primary 
concern to- thos^ of us interested Jji^^_,^x^nding tl^e concept of HRD) . 
• A second general concern about the approach described here' is how does it 
deri^ its name as a socio-technical continuum (or system). A t)est answer is 
based more oh personal experience than either theory or^the ^i^resent vogue of , 
quayty assurance In business and industry. Technical knowledge and ^kills/^ 
about hpw to conduc^ an evaluation cati be obtained by studying numerous 
excellent tpxts or by classroom study. Aiv. evaluator ' s knowledge and skill x 
applicatiorj, however, does nob necessarily yield a useful, successful 

evaluation. An additional essential "quality of advaiTc>ed social skills^, is 

♦ " i • " 

of significant importance. Included among these skills are being sensitive 

I* - 

to VariQus acfdiences <the willingness, to listen and interact), being intuitive, 
being responsibly in varioiis v/ays to various groups of people, being .tactful , 
the ability to anticipate questions especially with regard to management, a 
conibination instinct and expjerience, and being able to think on one's feet 
while talking. Thes^ skills dan not generally be learned through'books or - 
classrocmr experiei/ce. An optimal corabindtion of social and technical skills 

' ^ ■ I 

for evaluators should yield greater likelihood for a suocassful quality 

^ . / 1 ' 

assurance program, but would not nebessarily guarantee it. 



"^One of my coll*eagues pojLnted out, appropriately so, that an evaj.uator's . i 
'fimctional autht^ity*^ also has a great deal to do with evaluation success or impact. 
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A third general concern that readers may have a-cquired through this 

manuscript is the attention given to the utility of ^valuation results j 

(See grandenburgt 1980).' Utility or usefulness also has its social and 

^ • « 

technical sides whic4i are hopefully not incompatible. A number of references 

were made to some of the social aspects such as "volumes of data/' "lack of 

organizational knowledge*^ and lack. of social sophistication in the application' 

'of evaluation methodology in this mat(uscript. UTiy produce evaluation reports 

that people can not read nor even have any hope of interpreting the results? 

^ Why persist in obtsalning all the Information available on a given objective if 
nobody is interested in the final outcome? One suggestion given preventing 
this dilemma was to center the investigation around issues rather than 
objectiyes, i.e., focus on the major points of <:ontention to alleviate future* 
confronrtations. A second suggestion was to make use of *evaluators that have 

. 'advanced social as well as technical skills. To sum up/Alkin states "the 
^orientation of the evaluator is a deci'siv^ factor - perhaps the most 

influential - in determining whether utilization will occur" (Alkin and Law, 

• i 

1980, p. 79). 

On. the technical sXde of the utility issue, this writer woiTld like to see 
more attention given to Bayesian applications to cii*c1^ion-maki^^ in the quality 
assurance process. Bayesian analysis would -fit ^comfortably into the decision- 
% oriented studies' described by Stufflebeam and wibster-/ This type of analysis 
.IdnkS evaluation to planning and thus to recycling and feedback while also 
blinding judgment with objective evidence. Furthermore, it can ^n^wer the ' 
extent to which each decision al-ternalive maximizes the realization of issues 
'related to outcomes by ' pr'oviding utility information on each alter,native 
(Saar, 1980) • Although the statistical sophistication ijequired to implement 
Bayesian* analysis does crea^te som^ probl^ems, these are^^not insurmountable and 
may be very' well .worth the effort. , ' • / 
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FJ.gure 1^ ' * ^ 

General Context of Evaluation in Three Types of HRD Programs 
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General Framework:. 
Evaluation by Time Dime^idlons 
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* Figure 3 
■ .*■ Action Cothponent Subsystem 
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